Smart display

ABSTRACT

A smart display allows a user to build custom layouts of user interface blocks on the smart display independent of the software on the computer creating the user interface. A customization mechanism in the smart display allows a user to select portions of a user interface and move them to different positions on the display. The customization mechanism creates custom layout metadata that defines a screen offset for portions of a user interface moved by the user. The smart display monitors the incoming display data and re-assigns pixel rendering data to the new location in the moved user interface blocks as the data coming from the computer application changes.

BACKGROUND

1. Technical Field

This disclosure generally relates to computer displays, and morespecifically relates to a smart display that allows a user to reassignthe layout of the displayed content independent of the computer hardwareand software.

2. Background Art

A computer display or monitor is a well known and often an essentialpart of a computer system. The computer display or monitor provides avisual interface for a user to interact with what is going on inside thecomputer. While historically a computer display was typically a cathoderay tube (CRT), modern computer displays may also be one of manytechnologies including LCD (liquid crystal display), and LED (lightemitting diode). A computer display may also incorporate one of avariety of touch screen technologies.

Regardless of the underlying technology, the display screen typicallyallows the user to interact with one or more processes or applicationsexecuting on a computer. The application controls how information ispresented to the user in what is called a user interface on the display.With many applications, a user may be able to have some control over theuser interface but any such control is within the limits of the userinterface of the application.

BRIEF SUMMARY

A smart display allows a user to build custom layouts of user interfaceblocks on the smart display independent of the software on the computercreating the user interface. A customization mechanism in the computerdisplay allows a user to select portions of a user interface and movethem to different positions on the display. The customization mechanismcreates custom layout metadata that defines a screen offset for portionsof a user interface moved by the user. The smart display monitors theincoming display data and re-assigns data to the new location in themoved user interface blocks as the data coming from the computerapplication changes.

The foregoing and other features and advantages will be apparent fromthe following more particular description, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The disclosure will be described in conjunction with the appendeddrawings, where like designations denote like elements, and:

FIG. 1 is a block diagram of a computer system connected to a smartdisplay with a customization mechanism using custom layout metadata;

FIG. 2 is a block diagram that illustrates an example of custom layoutmetadata;

FIG. 3 is a block diagram that represents a user interface displayed ona touch screen of a smart display according to an example of anembodiment as described herein;

FIG. 4 is a block diagram that continues the example of FIG. 3;

FIG. 5 is a block diagram that continues the example of FIG. 4;

FIG. 6 is a method flow diagram for a customization mechanism thatallows a user to build a custom layout of user interface blocks on asmart display; and

FIG. 7 is an example of a method flow diagram for displaying a customlayout on a smart display.

DETAILED DESCRIPTION

Described herein is an apparatus and method for a smart display thatallows a user to build custom layouts of user interface blocks on thesmart display independent of and without the knowledge of the softwareon the computer creating the user interface. A customization mechanismin the smart display allows a user to select portions of a userinterface and move them to different positions on the display. Thecustomization mechanism creates custom layout metadata that defines ascreen offset for portions of a user interface moved by the user. Theportions of the user interface moved by the user are typicallyrectangular blocks. The smart display monitors the incoming display dataand re-assigns display data to the new location in the moved userinterface blocks as the data coming from the computer applicationchanges.

FIG. 1 illustrates a block diagram of a computer display system 100 asdescribed herein having a computer 110 and a smart display 112. Thecomputer 110 represents a common multi-purpose personal computer.However, those skilled in the art will appreciate that the disclosureherein applies equally to any computer system or electronic devicecapable of being connected to a display or monitor. As shown in FIG. 1,computer 110 comprises one or more central processing units (CPU) 114, amain memory 116, and a mass storage device such as a hard drive 118.These system components are interconnected through the use of a systembus (not shown). The computer 110 includes an operating system 120.Those skilled in the art will appreciate that the spirit and scope ofthis disclosure is not limited to any one operating system. Any suitableoperating system can be used. Operating system 120 is a sophisticatedprogram that contains low-level code to manage the resources of computer110. An application 122 executes on the CPU 14 from the memory 116 undercontrol of the operating system 120. The application 122 in conjunctionwith the operating system 120 provides data to a graphics processingunit (GPU) 126. The GPU 126 then provides display data to the smartdisplay 112. The GPU 126 provides a display data signal 124 to passdisplay data to the smart display 112 in a manner know in the prior art.As used herein, display data is data sent by the GPU to the display.Display data is typically pixel rendering data that includes pixelposition and color.

Again referring to FIG. 1 the smart display 112 inputs the display datasignal 124 with the display data from the GPU 126 in the computer 110.The smart display 112 includes a central processing unit (CPU) 128connected to a memory 130. The smart display further includes a pixelmemory 132 with an input pixel buffer 134 that holds a copy of thedisplay data coming from the computer for analysis, and a custom pixelbuffer 136 as described further below. The memory 130 includes acustomization mechanism 138 that is a software entity that creates andmaintains custom layout metadata 140 as described herein. The memoryfurther includes a border detection mechanism 142 for detecting borderchanges in the video data to determine when the user interface blockbeing re-assigned is no longer present in the display data. Display datareceived on the display data signal 124 from the GPU 126 is stored inthe input pixel buffer 134. The display data in the input pixel buffer134 is processed by customization mechanism 138 and the layoutcustomization logic (LCL) 144 to fill the custom pixel buffer 136. TheLCL provides logic to re-assign blocks of pixel data on the touch screen146. The custom pixel buffer 136 then supplies display data to a displayscreen, in this case a touch screen 146. Each of these entities isdescribed further below.

FIG. 2 is a block diagram that represents a highly simplified diagram ofcustom layout metadata 128. The custom layout metadata 128 includes auser interface ID 212 and may include other information regarding theoriginal layout 214 of the user interface for the correspondinginterface ID 212. The custom layout metadata 128 further includesreassignment data 216 and 218 that defines sections of the userinterface that have been specified by the user to be reassigned to a newlocation on the customized display. In the illustrated embodiment, theuser reassignment data includes a user assigned location for at leastone pixel block 216 and an offset 218 for each pixel block or sectionthat has been relocated in the custom layout. An example of this data isgiven in the example embodiments described below.

As mentioned above, the smart display 112 allows a user to build customlayouts of a user interface on the smart display independent of thesoftware on the computer creating the user interface. Independent of thesoftware as used herein means without the knowledge of or support of thecomputer operating system hardware and software. The smart displayutilizes a customization mechanism 138 (software) in the computerdisplay to allow a user to select portions or blocks of a user interfaceand move them to different positions on the touch screen 146 of thesmart display. Since the portions of the user interface are typicallyrectangular blocks of pixels, the terms “user interface blocks”, andpixel blocks are used herein, but it is understood that any shape couldbe selected and moved as described herein. A user interface block meansa portion of the user interface that the user can select to reassign toa new location on the custom user interface. As used herein, a pixelblock is the display data that is in the area or user interface blockselected by the user to be moved.

In response to the user actions to create a custom layout, the customlayout mechanism creates a record in the custom layout metadata for eachuser interface with a custom layout. After the custom layout is created,the smart display will reassign pixel blocks of the user interface anytime the same user interface and/or a user interface with the same basicpattern or layout is detected in the incoming display data signal 124. Arecord in the custom layout metadata may include a user interface ID212, an original layout 214, a user assigned location 216, and an offset218 as shown in FIG. 2. Again referring to FIG. 1, the customizationmechanism 138 monitors input display data in the input pixel buffer 134for user interface blocks that match records in the custom layoutmetadata. The border detection mechanism 142 monitors the input pixelbuffer 134 to find a user interface block and the customizationmechanism compares any user interface block found to records in thecustom layout metadata 140. Where there is a match, the customizationmechanism 138 supplies the offsets to the LCL 144 to reassign the pixelblocks identified by the user to modify the display data to build acustom layout using the custom pixel buffer 136. The custom pixel bufferthen sends the display data with the custom user interface layout to thetouch screen 146.

As discussed above, the pixel memory 132 has an input pixel buffer 134and a custom pixel buffer 136. Preferably the input pixel buffer 134 isan array of memory to hold pixel information for the current screen ofdisplay data coming from the computer system 110. This allows the borderdetection mechanism to scan the input pixel buffer for user interfaceblocks that are defined by the user and stored in the custom layoutmetadata 140. In a first example, the custom pixel buffer 136 is also anarray that holds display data for the current custom layout. The custompixel buffer 136 holds a copy of the input pixel buffer 134 as modifiedby the LCL to create the custom user interface layout of pixel blocksper the custom layout metadata 140. Thus in this example, the contentsof the input pixel buffer 134 are copied to the custom pixel buffer 136and then modified in the custom pixel buffer 136 by the LCL to bedisplayed in the custom layout on the touch screen 146.

Alternatively, the custom pixel buffer could be simply a latch thatholds a single pixel's worth of display data. In this case, the LCLwould detect when a pixel of data coming from the input pixel buffercorresponds to a location in the custom layout metadata 140 and applythe corresponding offsets in the custom layout metadata before beingsent to the touch screen 146. In this case the LCL will monitor eachpixel as it is loaded into the custom pixel buffer and change the pixeldata as needed before it is sent to the touch screen 146.

FIGS. 3, 4 and 5 illustrate an example of selecting user interfaceblocks and moving corresponding pixel blocks of display data on a smartdisplay. FIG. 3 represents a user interface 300 depicted on the touchscreen 146 as originally created by an application 122 executing on thecomputer system 110 (FIG. 1). In this example, the touch screen 146 is1024 by 768 pixels. For this example, we assume the pixels of the touchscreen 146 are arranged in a Cartesian coordinate system in the firstquadrant of the X,Y plane. The lower left hand corner of the touchscreen represents coordinates 0,0 and the upper right corner of thetouch screen represents coordinates 1024,768. Similarly, the upper lefthand corner of the touch screen represents coordinates 0,768 and thelower right corner of the touch screen represents coordinates 1024,0.

For the example illustrated in FIGS. 3-5, we assume that the applicationproduces a user interface on the touch screen with four different userinterface blocks—block 1 310, block 2 312, block 3 314 and block 4 316as shown in FIG. 3. The user interface blocks represent portions of theuser interface that are typically represented to the user as arectangular shape. The user interface blocks 310, 312, 314, 316 have aborder as shown by the outline of the blocks. Inside the borders, theuser interface blocks typically have data or a selection of pixelbuttons. For example, block 1 310 may be a control button area and block3 314 may be a text entry area. Any pixel data, including UI controlbuttons within the user interface block will be moved with the selecteduser interface block and are not shown in the Figures. The user (notshow) wishes to rearrange these blocks to create a custom layout usingthe smart display. In this example, the user identifies a pixel blockfor each of these user interface blocks to the smart display by“clicking” on opposing corners of the blocks. As used herein, the term“clicking” means using any input device for the user to gesture orindicate a screen location to the smart display. Any common input devicesuch as a mouse or touch screen can be used to identify the pixelblocks. The user can click on any two opposing corners of a userinterface block to identify a pixel block. For this example with a touchscreen, the user touches the lower left corner 318 and the upper rightcorner 320 to create a pixel block 1 corresponding to user interfaceblock 1 310. Likewise the user continues to identify each of the otherthree pixel blocks corresponding to user interface blocks 312, 314, 316.In this example, the user interface blocks and the corresponding pixelblocks use the same reference numbers for simplicity.

FIG. 4 continues the Example started with reference to FIG. 3. After theuser has identified the pixel blocks as described above, the user isthen able to click and “drag” the pixel blocks to the desired locationon the custom user interface 400. In this example, the user has finisheddragging block 1 310 to the bottom of the touch screen 146. Theremaining pixel blocks 312, 314 and 316 are shown in their originalposition and are partly covered by block 1 310. The user can continue todrag the pixel blocks to desired locations. FIG. 5 illustrates thecustom user interface 400 after the user has moved the remaining pixelblocks. As described above, in response to the user dragging pixelblocks the customization mechanism creates a record in the custom layoutmetadata. For the movement of user interface block 1, the customizationmechanism creates a pixel offset for the display that is part of thecustomization metadata. If we assume the coordinates of the originallocation of the lower left hand corner as 20,628, then the custom layoutmetadata for block 1 would be as follows.

X Y Original Location 20 628 User Assigned Location 20 20 Offset 0 −600

Using the above offset in the metadata for block 1, the layoutcustomization logic reassigns all pixels within the borders of block 1(between the lower left hand corner 318 and the upper right hand corner320) in the original layout to the area of the display shown by block 1310 in custom layout of FIG. 5. In this example, all the pixels in block1 are given an offset of −600 pixels. The custom layout metadata wouldinclude similar data for blocks 2, 3 and 4. As long as the same userinterface blocks are in the input data to be displayed the customizationmechanism applies the same custom layout metadata to the LCL. The borderdetection logic 142 (FIG. 1) can be used to determine when the displaydata changes such that the same user interface pattern with the customlayout is no longer being displayed. When this happens, thecustomization mechanism instructs the LCL to render the pixels of thedisplay using the original layout from the computer (the unmodified ordefault display data).

FIG. 6 shows a method 600 for a customization mechanism that allows auser to build a custom layout of user interface blocks on a smartdisplay. The steps in method 600 are preferably performed by thecustomization mechanism 136 (FIG. 1), but portions of the method mayalso be performed by other software associated with the smart display.First, display the existing or default user interface layout as receivedfrom the computer step (610). Next, enter into a customization mode,preferably through some action by the user on the smart display (step620). Then, detect a user designating pixel blocks for the custom layout(step 630). Then move the designated pixel blocks to locations specifiedby the user, preferably by allowing the user to drag the pixel blocks toa new location on the display (step 640). Then save custom layoutmetadata for the custom layout created by the user (step 650). Themethod 600 is then done.

FIG. 7 shows a method 700 for displaying a custom layout on a smartdisplay. Method 700 is preferably performed by the customizationmechanism in conjunction with layout customization logic (LCL) 132 (FIG.1), but portions of the method may also be performed by other softwareassociated with the smart display. Method 700 is performed for incomingdisplay data to detect and display custom layouts on the smart display.First, receive a display input from a computer (step 710). Next, detecta default pixel layout for a user interface in the incoming display data(step 720). Search for a corresponding custom layout pattern for thedefault layout (step 730). If a custom layout is not available (step730=no), then render pixels of the display using the original defaultlayout (step 750) and the method is done. If a custom layout isavailable (step 730=yes), then render pixels using the custom layoutusing the LCL and the custom layout metadata (step 760). The method isthen done.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device. A computer readablesignal medium may include a propagated data signal with computerreadable program code embodied therein, for example, in baseband or aspart of a carrier wave. Such a propagated signal may take any of avariety of forms, including, but not limited to, electro-magnetic,optical, or any suitable combination thereof. A computer readable signalmedium may be any computer readable medium that is not a computerreadable storage medium and that can communicate, propagate, ortransport a program for use by or in connection with an instructionexecution system, apparatus, or device. Program code embodied on acomputer readable medium may be transmitted using any appropriatemedium, including but not limited to wireless, wireline, optical fibercable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider). Aspects of the present invention are described below withreference to flowchart illustrations and/or block diagrams of methods,apparatus (systems) and computer program products according toembodiments of the invention. It will be understood that each block ofthe flowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks. These computer program instructions may also be stored in acomputer readable medium that can direct a computer, other programmabledata processing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks. The computer program instructions may also beloaded onto a computer, other programmable data processing apparatus, orother devices to cause a series of operational steps to be performed onthe computer, other programmable apparatus or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

One skilled in the art will appreciate that many variations are possiblewithin the scope of the claims. While the examples herein are describedin terms of time, these other types of thresholds are expressly intendedto be included within the scope of the claims. Thus, while thedisclosure is particularly shown and described above, it will beunderstood by those skilled in the art that these and other changes inform and details may be made therein without departing from the spiritand scope of the claims.

The inventions claimed is:
 1. A smart display comprising: a displayscreen connected to a pixel memory in the smart display, wherein thepixel memory includes an input pixel buffer and a custom pixel buffer,where the custom pixel buffer is directly connected to the displayscreen and supplies display data to the display screen; a processor anda memory in the smart display; a display data input of the smart displaythat receives display data from a single video source and places thedisplay data in the input pixel buffer, wherein the display data ispixel rendering data that includes pixel position and color with userinterface blocks having an original layout, wherein the display data isreceived from a graphics processing unit of a computer connected to thedisplay data input of the smart display; a customization mechanismstored in the memory of the smart display and executing on the processorof the smart display that creates custom layout metadata in response touser input for pixel blocks of the display data received from thecomputer connected to the smart display and placed in the input pixelbuffer; and layout customization logic in the memory of the smartdisplay that in conjunction with the customization mechanism uses thecustom layout metadata to reassign the received display data in theinput pixel buffer to a new location in the custom pixel buffer torelocate the pixel blocks to a custom layout on the display screen. 2.The smart display of claim 1 wherein the custom layout metadatacomprises data of the original layout of the user interface blocks, auser assigned location for at least one pixel block, and a pixel offsetfor the user assigned location of the pixel block in the custom layout.3. The smart display of claim 2 wherein the layout customization logicuses the pixel offset to reassign the at least one pixel block to theuser assigned location on the display screen.
 4. The smart display ofclaim 1 wherein the display screen is a touch display.
 5. The smartdisplay of claim 1 and wherein the customization mechanism of thedisplay screen allows a user to create custom layout metadata torelocate the pixel blocks to a custom layout by allowing the user toselect user interface blocks and drag corresponding pixel blocks to acustom location on the display screen.
 6. The smart display of claim 1wherein the custom pixel buffer is an array for holding custom displaydata for the display screen.
 7. The smart display of claim 1 wherein thecustom pixel buffer is a register for holding display data for a pixelof custom display data for the display screen.
 8. A computer-implementedmethod for a smart display, the method comprising the steps of:providing a display screen in the smart display connected to a pixelmemory in the smart display, wherein the pixel memory includes an inputpixel buffer and a custom pixel buffer, where the custom pixel buffer isdirectly connected to the display screen and supplies display data tothe display screen; receiving display data from a single video sourcewith pixel rendering data that includes pixel position and colorreceived from a graphics processing unit of a computer connected to adisplay data input to the smart display and placing the display datainto the input pixel buffer; displaying an existing user interfacelayout with user interface blocks on the display screen using thedisplay data; entering a customization layout mode in the smart display;detecting a user designating pixel blocks for the user interface blocksto create a custom layout; moving designated pixel blocks from the inputpixel buffer to a location indicated by the user in the custom pixelbuffer; and saving custom layout metadata for the custom layout of thepixel blocks designated by the user.
 9. The method of claim 8 whereinthe step of moving the designated pixel block further comprises allowingthe user to drag the designated pixel blocks to a new location on thedisplay screen.
 10. The method of claim 8 further comprising the stepsof: receiving a display input; detecting a default layout of userinterface blocks; searching for a corresponding custom layout for thedefault layout; and where there is a custom layout available for adetected default layout, rendering pixels on the display screen usingstored custom layout metadata.
 11. The method of claim 10 wherein thecustom layout metadata comprises data of the original layout of the userinterface blocks, a user assigned location for at least one pixel block,and a pixel offset for the user assigned location of the pixel block inthe custom layout.
 12. A computer-implemented method for a smartdisplay, the method comprising the steps of: providing a display screenin the smart display connected to a pixel memory in the smart display,wherein the pixel memory includes an input pixel buffer and a custompixel buffer, where the custom pixel buffer is directly connected to thedisplay screen and supplies display data to the display screen;receiving display data from a single video source on a display datainput to the smart display, wherein the display data is pixel renderingdata that includes pixel position and color received from a graphicsprocessing unit of a computer connected to the smart display and placingthe display data into the input pixel buffer; detecting a default layoutof user interface blocks in the display data; searching for acorresponding custom layout for the default layout; and where there is acustom layout available for a detected default layout, rendering pixelson the smart display using stored custom layout metadata by movingdisplay data from the input pixel buffer to a new location indicated bythe custom layout in the custom pixel buffer; where there is not acustom layout available for a detected default layout, rendering pixelson the display screen using the default layout.
 13. The method of claim12 wherein the custom layout metadata comprises data of the originallayout of the user interface blocks, a user assigned location for atleast one pixel block, and a pixel offset for the user assigned locationof the pixel block in the custom layout.